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Background / Context: 

For intervention studies involving binary treatment variables, procedures for power 
analysis have been worked out and computerized estimation tools are generally available. Of 
greatest importance for educational research, such procedures and tools are available for the 
kinds of complex multilevel designs that are often required for testing education interventions, 
for example, Bloom (1995, 2005, 2006), Hedges & Rhoads (2010), Konstantopoulos (2008a, 
2008b, 2009, 2010, 2012), Murray (1998), Raudenbush (1997), Raudenbush & Liu (2000), 
Raudenbush, Martinez, & Spybrook (2007), and Schochet (2008). Among the excellent 
computer programs available for conducting power analysis for cluster randomized experiments 
are Optimal Design (Raudenbush, Spybrook, Congdon, Liu, & Martinez, 2011), CRT-Power 
(Borenstein & Hedges, 2012), and PowerUp! (Dong & Maynard, 2013). 

However, there are relationships other than the main effects of binary treatment variables 
that interest education researchers. For example, researchers may wish to detennine if some 
classroom practice described by a continuous variable, such as the number of disruptive incidents 
or the amount of time on task, is related to student academic achievement when other influential 
classroom characteristics are statistically controlled. Or, within the context of an experimental 
study, researchers may wish to determine whether the effects of the intervention differ 
conditional on moderator variables such as pretest, ethnicity, school climate, or the fidelity of 
implementation. Power analysis for relationships between continuous predictors and dependent 
variables in multivariate, multilevel models cannot be accomplished with the procedures and 
estimation tools that have been developed for intervention studies. Moreover, power for 
moderator analyses in multilevel intervention studies cannot be estimated using the procedures 
and tools for estimating the power of main effects. Compared to power analysis for main effects 
in cluster randomized experiments, there is less support for power analysis for relationships 
involving continuous predictors or the interactions of moderator variables with treatment effects. 
The only research we have identified on power analysis for relationships of continuous 
predictors with outcomes in a multivariate multilevel model is Snijders & Bosker’s (1993), 
which was restricted to a two-level HLM, and did not result in any computational tools for use 
by researchers. For moderator relationships in experimental studies, Bloom (2005) and Spybrook 
(in press) have presented procedures for conducting power analysis for binary moderators in 
two- to four-level cluster randomized experiments, but have not extended those procedures to 
include continuous moderator variables. Most recently, Mathieu, Aguinis, Culpepper, & Chen 
(2012) conducted a comprehensive Monte Carlo simulation to estimate the statistical power to 
detect cross-level interaction effects. However, Mathieu et al (2012) only studied two-level 
analysis without including covariates, and did not provided closed fonn formulas to estimate the 
statistical power, minimum detectable effect size, or minimum required sample size to detect 
meaningful effects. 

Purpose / Objective / Research Question / Focus of Study: 

The purpose of this study is to: (1) develop the statistical fonnulations for calculating 
statistical power, minimum detectable effect size (MDES) and its confidence interval, and 
minimum required sample size to detect the effects of a continuous moderator variable at level 1 
or level 2 in two-level simple cluster random assignment designs, and (2) operatize these 
formulas in the enhanced version of PowerUp! (Dong & Maynard, 2013) to create spreadsheets 
for calculating MDES, etc. 

Significance / Novelty of study: 
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Educational researchers have interests in the effects of continuous moderators in cluster- 
randomized experiments. Statistical power analysis is appropriate in the planning stages to help 
researchers design studies with sufficient power to detect such relationships when they are large 
enough to have practical or theoretical significance. However, currently there is no tool available 
for researchers to conduct such power analysis. This study will provide a tool for power analysis 
to detect the continuous moderator effect in two-level simple cluster random assignment designs. 

Research Design: 

Framework 

We use the framework of the minimum detectable effect (MDE) popularized by Bloom 
(1995, 2005, 2006; Murray, 1998). Although this framework was originally developed for 
estimating the power for detecting the relationship between a binary treatment variable and an 
outcome variable, it can be extended to other relationships with dependent variables in multilevel 
models. MDE measured in raw scale units can be expressed as MDE(b 0 ) = M v SE(b 0 ) , where b 0 
is the effect (unstandardized coefficient) of the focal predictor, SE(b 0 ) is the standard error of 
that effect, and M v is a multiplier that carries information about the selected alpha level, 
statistical power target, and degrees of freedom for the significance test. Specifically, M v is the 
sum of two /-statistics. For one-tailed tests, M v = t a +t x _ p with v degrees of freedom (a function 
of sample size and number of covariates), and for two-tailed tests, M v = t a!2 + t x _ p ■ MDE, in turn, 
is the minimum effect (b 0 in raw scale units) that can be detected at the a level with probability 
(statistical power) 1 - J3 . 

When the focal predictor is a continuous variable, defining the effect size as the 
standardized mean difference is not appropriate. For the derivations to be developed in this 
project, we will use the standardized regression coefficient ( J3 0 ) for the predictor of interest as an 

effect size. This can be estimated using HLM by first standardizing the outcome variable and the 
predictors, i.e., iV(0,l), as the effect size metric. Note that this standardized coefficient is equal to 
the Pearson correlation (r) of the predictor and the outcome when there is only one predictor in 
the model and it is a semipartial correlation coefficient when there are two or more predictors. 

An alternate representation of the effect for the designs and analyses of interest here is 
the correlation between the predictor and the dependent variable. In multilevel analysis, 
however, there are multiple possible expressions of this correlation. For instance, in a two-level 
analysis with only one level-2 predictor, the correlation of the predictor and the individual (level 
1) outcome values (which is the same as the standardized regression coefficient), and the 
correlation of the predictor and the cluster means on the outcome variable at level 2 are different, 
but both represent the association of the predictor and the outcome and both can serve as an 
effect size metric. For some derivations, it is convenient to represent the effect in terms of the 
correlation at the level of the focal predictor (adjusted as needed for additional covariates in the 
model), but this can be easily converted to the corresponding standardized regression coefficient. 

Similarly, in the analysis of the effect of the moderator in multilevel experiments, the 
standardized coefficient of the interaction tenn for the treatment variable and the moderator is 
the effect size of primary interest. This coefficient can also be expressed in correlational terms as 
a function of the separate correlations between the predictor and the outcome for the control and 
treatment groups. The standardized coefficient and the difference between the treatment and 
control correlations (adjusted for other covariates as needed) are alternate representations of the 
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interaction effect and one can be converted to the other. 


As is typical in multilevel power analysis, we assume that the data are balanced such that 
each cluster has the same number of observations (n). We also assume that each cluster has the 
same values of the predictors and the same sampling variance for the coefficient of the predictor. 
Under these assumptions, the unique, minimum-variance, unbiased estimator of the coefficient ( 
b 0 ) of the focal continuous predictor would be the OLS (ordinary least square) regression 
estimator (Raudenbush & Bryk, 2002, p.43). Hence, we will use the OLS estimators to: (1) 
derive formulas for the estimate of the coefficient ( b 0 ) of the focal continuous predictor or the 
interaction tenn for a moderator relationship expressed by the correlation (r) or the standardized 
coefficient ( /?„ ), and (2) derive formulas for the estimate of the standard error ( SE(b 0 ) ) of the 

coefficient expressed as the correlation (r) or the standardized coefficient ( [i {) ). Furthennore, the 
minimum detectable effect should satisfy M v = MDE(b 0 )/ SE(b 0 ). The multiplier can be 
calculated based on the desired a level, statistical power 1 - J3 , sample size, and the number of 
covariates. Hence, the correlation (r) or the standardized coefficient ( /? 0 ) can be expressed as a 
function of the multiplier, M v , and the sample size, etc. 

We illustrate the procedures (but omit some details due to page limitation) for the 
derivations of MDES with two analytic examples. The first example is a basic two-level 
hierarchical linear model (HLM) that includes one level-2 focal continuous predictor, W, and no 
covariates. The second example is a two-level cluster randomized design with the sample equally 
divided between the treatment and control groups and a continuous moderator variable at level 2. 
The procedures illustrated for these two examples will be extended to the other design and 
analysis models for which we propose to develop power analysis fonnulations with adaptations 
that take the greater complexity of those designs into account. 

Two-level HLM with a level-2 continuous predictor 

The HLM including one level-2 continuous predictor, W. , is: 

Level 1: ^ ~ N(0 ,ct 2 ) 

Level 2: fi 0j = y 00 + y 0l Wj + u 0j , u 0j ~ N (0, ) 


Based on results from Raudenbush and Bryk (2002, pp. 39-41), we can derive MDES in 
terms of the correlation of the sample cluster mean, F . , and the level-2 predictor, W j , for a given 

level-2 sample size, J, desired a level, and statistical power ( 1 - (3 ) as: 


(1) MDES(r v J) = 


m; 

Ml +J-2 


where, M v =t a + t x _ p for one-tailed tests with v degrees of 


freedom (v = J -2 when there is only one predictor), and M v = t al2 + t x _ p for two-tailed tests. 
The MDES in terms of the standardized coefficient can be derived: 

— 2 — v — ~ y!p + (\-p)ln , where p is the unconditional intra-class 
^ l J 2 

correlation (ICC). 

Two-level cluster randomized design with a continuous moderator at level 2 


SREE Spring 2014 Conference Abstract Template 


4 




The full HLM for this example, including one treatment variable, T . , and one level-2 
moderator, W. , is: 

Level 1: Y tj = /J t)j + r (j , r r N(0,a 2 ) 

Level 2: f3 0J = / 00 + Yox Wj + y 02 Tj + Yo?,( w j x T , ) + u 0j , u 0j ~ N (0, r^ T ) 

The interest for moderator analysis is whether the parameter, ;/ 0 ,, which indicates the 
relationship between the treatment effect and the moderator, is statistically significant. 

Based on the previous results for a continuous level-2 predictor (Expressions 1 & 2), the 
minimal detectable effect sizes in terms of the incremental Cohen’s /and the correlation ( r' v ) of 


Y_j and W j for the treatment group can be calculated from Expressions 3 and 4: 



where M v = t a + t x _ p for one-tailed tests with v degrees of freedom ( v = / - 4 when HLM 
includes the treatment variable, moderator, and the interaction tenn for the treatment and 



Note that the original Cohen's/is one metric of effect size for OLS regression. We have 
two Cohen's fs for the control and treatment groups. It is the incremental Cohen's/(that is, 

/ - /. ) that represents the effect size of the moderator effect, i.e., the effect difference of the 

predictor between the treatment and control groups. The above results suggest that the 
incremental Cohen's/ due to the intervention, and the sample size ( J) must be large enough to 
satisfy Expression 4 to detect a significant differential effect (y 03 ) at the a level with statistical 
power of 1 — p . 

Results and Conclusions: 

This abstract only shows the preliminary results of the MDES for detecting a level-2 
continuous moderator effect in two-level cluster randomized experiments. The formulas for 
statistical power and the minimum required sample sizes will be derived accordingly. 
Furthennore, the MDES etc. for a level- 1 continuous moderator and with covariates in two-level 
cluster randomized experiments will be derived. All these formulas can be operated in Microsoft 
Excel in the enhanced version of PowerUp! to help researchers with designing moderation 
analysis in multilevel experiments. 
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